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Abstract 
This study focuses on the implementation of an early warning indicator and intervention system 
in 20 southern high schools. This model included a team of teachers, counselors, and student 
support services personnel who analyzed ninth grade student-level data and implemented and 
monitored interventions. The team was led by a half-time coach who established protocols for 
team operations and provided updated early warning indicator data. Findings indicated 
variability in the frequency and scale of the interventions initiated and monitored by the 
promotion coaches and early warning indicator teams. This study further demonstrates that 
higher levels of implementation of early warning and intervention components in the treatment 
schools were associated with improved levels of ninth grade student attendance and course 


performance even after controlling for prior attendance. 
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Implementation of an Early Warning Indicator and Intervention System 

Although high school graduation rates across the United States have risen for all students, 
including low-income and minority students, ethnic and income disparities persist. Compared to 
the national overall graduation rate of 82.3%, sixteen states have graduation rates below 70% for 
African American and low-income students, and 11 states have graduation rates below 70% for 
Hispanic students (DePaoli, Balfanz, & Bridgeland, 2016). Lower rates of graduation for these 
groups have been linked to higher levels of disengagement from school, as measured by 
attendance, behavior, and course performance indicators (Kemple, Segeritz, & Stephenson, 
2013). These indicators have been shown to predict future dropout as early as sixth grade 
(Balfanz, Herzog, & Maclver, 2007; Neild, 2009; Neild & Balfanz, 2006). Student behaviors 
during the ninth grade year in particular have been shown to be predictive of subsequent dropout. 
Early research revealed that failing no more than one semester of a core course in ninth grade 
and earning credits necessary for promotion to tenth grade predicted graduation with 80% 
accuracy. This on-track measure was even more predictive than test scores (Allensworth & 
Easton, 2005, 2007) and other student background factors (Allensworth, 2013). 

Part of the reason why ninth grade performance in particular is so predictive of dropout is 
that both engagement and achievement declines over the ninth grade year result in failure to earn 
course credits required for graduation. Allensworth (2013) noted that these declines were not 
due to the greater difficulty of ninth grade work compared to eighth grade work, but were instead 
related to a decline in ninth graders’ study habits and attendance. Interviews with ninth grade 
students suggested that these declines were related to students’ receiving less adult monitoring of 
behavior and less adult support in high school compared to middle school. Without the oversight 


of caring adults, ninth grade students took advantage of increased freedom to attend less often 
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and put in less effort. It was clear that ninth grade students needed more systematic monitoring 
and encouragement to ensure they attended school every day and did the work needed to pass 
their courses and earn the credits required to stay on track to graduation. 

Early Warning Systems 

In response to research on attendance, behavior and course performance as early warning 
indicators of students being off-track to graduation, much attention was given nationwide to the 
development of early warning systems (e.g., Dynarski et al., 2008; Pinkus, 2008; Therriault, 
O’Cummings, Heppen, Yerhot, & Scala, 2013). Over half of public U.S. high schools (U.S. 
Department of Education, 2016) now implement some type of early warning system using early 
warning indicator data to identify students who are at risk of dropping out of school. According 
to data gathered in a national survey on Early Warning Systems, data commonly used by most 
schools include attendance and truancy records, course grades, and disciplinary incidents (U.S. 
Department of Education, 2016). Other data less commonly used include state achievement test 
scores, homelessness, involvement with the criminal justice system, and indicators of limited 
English proficiency. Those who monitor these data are primarily school administrators and 
guidance counselors. Almost half of responding school leaders reported monitoring these data 
weekly. 

Although early warning systems are helpful for identifying students who have fallen off 
the path to graduation, they do not necessarily provide a structure for getting students back on 
this path. Research teams have recently examined early warning and intervention systems that 
include both early warning indicators to identify students and a process for matching, conducting, 


and monitoring student interventions (Davis, Mac Iver, & Fox, 2017; Faria et al., 2017). 
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One of these systems, the Early Warning Intervention and Monitoring System (EWIMS), 
is a seven-step process that includes establishing roles and responsibilities of team members, 
using an early warning data tool provided by the project, reviewing early warning data, assigning 
and providing interventions, monitoring interventions, and evaluating and refining the early 
warning process (Faria et al., 2017). A technical assistance liaison helps treatment schools 
establish an EWIMS team, with one person responsible for using the early warning tool and three 
to five other school staff to implement interventions. After a random assignment of 73 American 
high schools to either the seven-step process or business as usual, the research team examined 
differences in attendance, behavior, and course performance between ninth and tenth grade 
students in treatment and control schools. The results showed significant differences between 
the treatment schools and control schools with regard to chronic absences and course failures, 
with the treatment school students having fewer absences and failures. 

Although the early warning and intervention system was found to have a significant 
impact, the authors of the report stated that schools found implementation in their first year 
difficult (Faria et al., 2017). Seven of the 37 treatment schools stopped implementing the 
program during the study year. Attendance at implementation strategy training sessions declined 
over the year as well, from 97% of schools participating in the first regional training to 59% 
participating in later trainings. Of the 37 schools, only two schools achieved moderate or high 
levels of implementation of all seven steps. Further findings suggest that school staff had 
difficulty using the early warning data tool, did not have a consistent staff person to extract the 
data from the tool, and did not assign all students with early warning indicators to an 
intervention. The authors suggested that higher implementation could possibly have been 


realized if each school had been in the project for more than one year. 
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Another early warning and intervention system, the “Early Warning Intervention (EWI) 
Team” model, developed at the Center for Social Organization of Schools at Johns Hopkins 
University, is increasingly being employed by low graduation rate high schools and their feeder 
middle schools as part of the Diplomas Now (DN) School Transformation/ Turnaround model 
(Corrin et al., 2014, 2016; Sepanik et al., 2015). In comparison to “case management” 
approaches such as “Check and Connect” (Sinclair, Christenson, Evelo, & Hurley, 1998; 
Sinclair, Christenson, Lehr, & Anderson, 2003; Sinclair, Christenson, & Thurlow, 2005) or the 
graduation coach model (summarized in Georgia Department of Education, 2009; SERVE, 
2008), in which an individual has a specific role of providing or organizing direct supports to 
students identified as off-track to high school graduation, the EWI team model engages all 
important school adults, both academic and non-academic, who are in contact with a potential 
dropout, and provides guidelines and training in a sequenced approach, with frequent monitoring 
and revision, as a high school dropout prevention strategy. Similar to the EWIMS team (Faria et 
al., 2017) the EWI team approach shifts responsibility from an individual adult to a team of 
adults in a collective and collaborative approach joining academic and non-academic functions. 
One coordinator (the on-site graduation facilitator) works closely with school administrators to 
organize teams of teachers and adults providing supports to off-track students, but the 
responsibility rests with the teams rather than a single individual. Examining data for the same 
students across different classrooms, as occurs in the team meeting approach, has been shown to 
create an atmosphere of shared responsibility (Allensworth, 2013). Teacher teams focused on 
data across classrooms share strategies that are working for students in one class, such as 
English, that could be applied in another class, like math. Teachers can share information about 


struggling students and work together on a plan for each student across all of his or her classes. 
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In a prior study, Davis, Herzog, & Legters (2013) visited ten DN schools in seven cities 
across the United States, observed 13 team meetings, and conducted 75 team member interviews. 
Findings indicated that typical DN team meetings were 30-60 minutes in length, held bi-weekly, 
and were facilitated by the on-site graduation facilitator. The typical team meeting started with a 
quick follow-up discussion of students discussed in the previous team meetings. This discussion 
was followed by the team working through a focus list of about 5 to 11 students with early 
warning indicators prepared by the on-site graduation facilitator prior to the meeting. Team 
member interviews revealed challenges with the team meeting approach that included scheduling 
time for team meetings, absenteeism of key members, and occasions when the conversation 
about one student took up too much meeting time. Despite these challenges, the teachers were 
quite positive about the team meetings, stating that their teams communicated well, shared 
information readily, and were solution-focused. 

Study Overview 

The current study examines a version of the EWI Team model that does not include the 
additional staff and supports included in the DN model (professional development supports, 
additional staff from the Americorps-type City Year program, and a social worker from 
Communities in Schools), as well as a full-time school transformation facilitator to organize the 
EWI Team and interventions (Corrin et al., 2014; Corrin, Sepanik, Rosen, & Shane, 2016). In 
this more modest EWI Team model, the only level of support provided to schools was a part- 
time promotion coach focused on implementing the model in the school. The goal of the study 
was to examine how levels of implementation in this more modest EWI Team model are 
associated with ninth grade student outcomes. 


This study is part of a larger randomized control study of this EWI team-based approach 
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with 41 schools across a southern state conducted to determine whether the intervention 
improved ninth grade student attendance and course performance (Davis et al., 2017). To be 
eligible for participation in the study, schools had to have at least 100 ninth grade students and a 
graduation rate below 75%. All eligible schools within the state were contacted through their 
local district, and an initial pool of 77 schools agreed to participate. A researcher unrelated to the 
development of the program was asked to form 20 matched pairs (with one group of three, a 
treatment school matched to two control schools) according to school size and graduation rate. 
The goal was to obtain the closest matches possible from among the eligible schools. Following 
the initial matches, the researcher made adjustments to improve the balance on school-level 
variables (percentage of African American students and percentage eligible for free or reduced 
price lunch). She then randomly assigned the schools to the treatment condition (having a grant- 
funded half-time promotion coach for two years) or to the control condition (receiving a $5000 
payment each year for two years). The promotion coach “treatment” was expected to ensure that 
the school was 1) monitoring ninth grade student-level data on attendance, behavior, and course 
performance at least weekly; 2) holding meetings of ninth grade teachers and other staff 
members at least bi-weekly to discuss early warning indicator data and design interventions; and 
3) providing students exhibiting early warning indicators with appropriate interventions. 

As previous research has indicated, implementation of early warning systems with 
fidelity can be difficult (Faria et al., 2017). Further, scaling back the DN model to a part-time 
promotion coach represents a significant reduction in the supports provided to the school. 
Compared to a full-time DN facilitator with the additional DN supports, the part-time promotion 
coach would have significantly less resources and time to support implementation of the early 


warning intervention system. In this article we examine data gathered from the treatment schools 
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to assess the extent to which promotion coaches implemented early warning and intervention 
components and guided interventions in their schools during a school’s second year in the 
program. In addition, we analyze to what extent intervention implementation levels were related 
to student outcomes. Questions guiding our study included: 

1) What percentage of promotion coaches and Early Warning Indicator Teams met the 
thresholds for each characteristic of an early warning system (holding bi-weekly 
meetings, connecting with teachers, accessing and analyzing student data, and leading 
class-wide and student-based interventions)? 

2) To what extent was meeting these thresholds different for promotion coaches who 
worked less than 20 hours per week, 20 hours per week, or more than 20 hours per week? 

3) To what extent did promotion coaches and Early Warning Indicator Teams’ 
implementation of these practices predict students’ attendance and course passing? 

Method 

Participants 

The 20 treatment schools were located in 15 districts throughout the state, and ranged in 
size from about 350 to 2,350 students. Five of the schools were in urban (mid-size city) districts, 
three in small cities or suburbs, five in towns, and seven in rural areas. The percentage of 
minority students in treatment schools ranged from 1% to 100% (five schools had 97% or more). 
The free/reduced price lunch rate ranged from 41% to 90%. 

In the 20 treatment schools a half-time staff member was hired and assigned as a 
“promotion coach.” There were 22 coaches over the two-year time span due to one coach who 
was placed at two successive schools and three coaches who had to be replaced after the first 


year. Of these 22 coaches, 11 were recommended by their particular school or district and 11 
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were recommended by the state facilitator at the department of education. These 22 coaches had 
previously served in one or more positions related to education, including as a teacher (math, 
science, social studies, reading, language arts, art, technology, drama, special education), school 
principal, assistant principal, assistant superintendent, assessment administrator, adjunct 
professor, chief academic officer, student service coordinator, career coach, school counselor, 
mental health case manager, at-risk instructor or coordinator, school improvement facilitator, 
STEM director for the district, family intervention specialist, child welfare worker, academic 
coach, or media specialist. Many had held more than one of these positions. All candidates had 
to apply for the position and be interviewed by both the state facilitator and University team 
member. 

Promotion coaches received three days of training at the start of the project, four 
additional one-day trainings during the year, and bi-monthly visits from the state facilitator and 
implementation leader throughout the project. Implementation was rolled out in phases, with 10 
schools (Cohort 1) beginning in 2012-13 and 10 more schools (Cohort 2) beginning in 2013-14. 
During the first year of the project, Cohort 1 coaches had been recruited and received training by 
November 2012 and began working in schools generally by December 2012. Cohort 2 coaches 
had been recruited and received training the summer of 2013 and began working with their 
schools in August 2013. Implementation occurred over two school years for each cohort, with a 
shorter implementation time during the first year for Cohort 1. Our pre-specified analysis plan 
focused only on the second year of implementation for each cohort because of anticipated delays 


in getting implementation underway during the first year. 
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Coaching Responsibilities 

Promotion coaches were half-time employees, although promotion coach schedules 
varied by school. Most coaches were at their school two to three days per week, for a total of 20 
hours per week on average. Because these were additional staff members for the schools, they 
were not supposed to have any other responsibilities at the school other than those outlined 
below. Their responsibilities were aligned to characteristics of the “Early Warning Intervention 
(EWI) Team” model, developed at the Center for Social Organization of Schools at Johns 
Hopkins University. These characteristics, listed below, provided a focus for the summer 
promotion coach training. 

1. Promotion coaches were expected to hold biweekly early warming indicator (EWI) team 
meetings throughout the school year or at least every two to three weeks. 

2. Coaches were encouraged to meet with teachers individually outside of the team 
meetings to provide support and encouragement related to implementing the interventions 
determined during the team meetings. 

3. Coaches were to have administrator level access to data in order to present student level 
data on attendance, behavior, and course performance at the early warning intervention 
team meetings. 

4. Coaches were expected to examine student level data at least weekly to identify trends 
and to have the data ready to present at the early warning intervention team meetings. 

5. Promotion coaches were to lead the EWI team on implementing student interventions. 
Interventions used by teams fell into one of seven different categories including four 
types of grade-wide interventions (professional development for teachers, classroom 


talks, report card conferences, attendance initiatives); small group interventions; and two 
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types of individual student interventions (talks with parents and talks with students). 
Data 

Implementation data were collected through two main sources: a weekly log of activities 
kept by each promotion coach and submitted to program developers, and hour-long interviews of 
promotion coaches conducted at the end of each year. The weekly logs kept by the coaches 
included a list and description of their initial and follow-up activities, their thoughts and 
reflections, a summary of the “challenge of the week,” their perceptions of the support they 
needed, and details relating to any EWI meetings held that week, including the time and date of 
the meeting, number of students discussed, materials used, list of attendees, list of interventions 
discussed or assigned, and identification of focus students’ early warning indicators, whether 
poor attendance, behavior, and/or course performance. 

Interviews included questions pertaining to the initial and subsequent follow-up training 
from the developer team, data access and management (8 items); EWI team meetings and 
interventions (6 items); and lessons learned (5 items). These questions are provided in the 
Appendix. 

Administrative student data were obtained from each district, including outcome 
measures of attendance and course grades for the ninth grade year as well as demographic and 
status background variables and a measure of prior year’s attendance. Data on school level 
characteristics were obtained from the Common Core of Data (2013-2014 school year). 
Measures 

Each school was assigned a score on each of 11 binary fidelity thresholds listed in Table 
1. These 11 scores aligned to characteristics of the “Early Warning Intervention (EWI) Team” 


model, developed at the Center for Social Organization of Schools at Johns Hopkins University. 
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The threshold indicating whether or not the coach met a certain level of activity within each 
category was determined based on the early warning intervention model, with a 0 indicating that 
they did not meet the threshold and a 1 that they did meet the threshold. Scores on these 
thresholds were determined through analysis of data from the coach interviews and weekly logs. 
Details for each threshold are summarized in Table 1. 

Interrater agreement. Two raters were used to assess interrater agreement. First the 
raters coded one randomly selected school out of the twenty treatment schools and coded the 
coach at the school on each of the 11 thresholds listed in Table 1. The coders met again and 
adjusted the coding sheet for codes that were rated differently. The raters did this process with 
three more coaches. The raters indicated the same code for these four coaches 83% of the time. 
Finally, the raters individually rated a set of 5 coaches. The raters indicated the same code for 
these four coaches 89% of the time. Details on what constituted meeting each threshold can be 
found in Table 1. 

Composite fidelity scores. Three composite scores were calculated. First, total Early 
Warning System (TOTAL) composite fidelity score was calculated as the average of the 11 
different threshold scores (items 1-11 from Table 1). A Team and Data (TEAM) composite 
fidelity score was calculated as the average of the four early warning team and data thresholds 
(items 1-4 from Table 1). A Use of Interventions (INTERVENTION) composite fidelity score 
was Calculated as the average of the final seven threshold items (items 5-11 from Table 1). 

Coach rating on proportion of time devoted to EWI team model duties. Based on 
site visits and interviews with coaches, the project field manager rated each coach on whether the 
time each devoted to specified duties was below the expected 20 hours per week, at the expected 


level, or above the expected level. 
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Student attendance. The number of days each ninth grade student attended was divided 
by the number of school days for their district to obtain an attendance rate. In addition to the 
attendance rate, two thresholds were examined. Dichotomous variables were created for at least 
95% attendance rate and for at least a 90% attendance rate. Students’ previous year’s attendance 
rate (eighth grade attendance for first-time ninth graders; previous year’s ninth grade attendance 
for repeat ninth-graders) was used as a covariate in the data analysis. 

Course passing. Course grades were included in student administrative data. The 
percentage of courses passed was obtained by dividing the number of courses passed by the 
number of courses taken. In addition, two thresholds were examined: passing all ninth grade 
classes (vs. failing one or more classes), and failing no more than one class (vs. failing more than 
one class) 

Data Analysis 

Descriptive statistics. The first research question was, “What percentage of promotion 
coaches and Early Warning Indicator Teams met the thresholds for each characteristic of an early 
warning system (holding bi-weekly meetings, connecting with teachers, accessing and analyzing 
student data, and leading class wide and student based interventions)?” To address this question, 
the number of coaches meeting each of the 11 thresholds was examined. In addition, the average 
score across all of the Early Warning System characteristics, the four characteristics related to 
team meetings and data, and the seven characteristics related to student interventions were 
examined. Further, correlations and scatterplots using SPSS software were conducted to 
determine the relationships among these three average scores. 

The second question was, “To what extent was meeting these thresholds different for 


promotion coaches who worked less than 20 hours per week, 20 hours per week, or more than 20 
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hours per week?” To address this question we examined the percent meeting each threshold for 
the 11 thresholds and three total scores. Using SPSS software, we conducted a one-way analysis 
of variance to determine if there were significant differences between schools with a coach 
supporting the EWS teams less than 20 hours per week, 20 hours per week, or more than 20 
hours per week. We used Fisher’s Least Significant Difference in order to determine which of 
the three groups were significantly different from each other. 

Multilevel models. The third research question was, “To what extent did promotion 
coaches and Early Warning Indicator Teams’ implementation of these practices predict students’ 
attendance and course passing?” To address this question we used multilevel logistic regression 
and hierarchal linear modeling using HLM6 software (Raudenbush, Bryk, & Congdon, 2004). 

Using logistic regression, each total score (three scores at the bottom of Table 2) was 
used to predict two ninth grade attendance indicators (at least 95% attendance or at least 90% 
attendance) and two course performance indicators (passing all classes, failing no more than one 
class) while controlling for the previous year’s attendance. 

Mathematically, we specified the model as follows: 
Level-1 Model (Student) 
Prob (Yi=1 | Bj) = Bi 
log[@ij/ (1- Gij)] = ni 
Niji = Boj + Ba*(Xiy) 

Yij = attendance or course performance score of student i in school j 

Xij = previous year attendance of student i in school j 
Level-2 Model (School) 


Boj = yoo + you*(Ej) + Uoj 
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Bij = y10 
Level-1 variance — 1 / [Oi (1 — Gij)] 

Ej = total, team, or intervention score of school j 

At level 1 (student level), we specified the models to predict binary variables for 
attendance and course failure (Yij) controlling for the student attendance in the previous year 
(Xij). At level 2 (school level), we included (in separate analyses for each predictor) the school 
implementation score for the EWI team components, the EWI intervention components, or the 
total combined implementation score (Ej). The model for predicting the dichotomous measures 
using the log odds of students being chronically absent or failing one or more courses included a 
constant (Boj at level 1, yoo at level 2), a coefficient term estimated for each variable (B1; at level 
1, yo. at level 2), and an error term at level 2 (uoj). 

In addition, the total, intervention, and team scores were used in a hierarchical linear 
model to predict continuous variables of ninth grade attendance (percent attendance) and course 
performance (percent of credits earned) while controlling for previous year attendance. 

Mathematically, we specified the model as follows: 

Level-1 Model (Student) 
Yij = Boj + Bajy*(Xi) + Hi 

Yij = attendance or course performance score of student i in school j 

Xij = previous year attendance of student i in school j 
Level-2 Model (School) 

Boj = yoo + yor*(Ej) + uoj 
Bij = y10 


Ej = total, team, or intervention score of school j 
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At level 1 (student level), we specified the model to predict attendance and course failure 
(Yij) controlling for the student attendance in the previous year (Xij). At level 2 (school level), 
we included (in separate analyses for each predictor) the school implementation score for the 
EWI team components, the EWI intervention components, or the total combined implementation 
score. The model equation included a constant (Boj at level 1, yoo at level 2), a coefficient term 
estimated for each variable (B1j at level 1, yo1 at level 2), and an error term (rij at level 1 uoj at 
level 2). 

Results 
Promotion Coach Use of Early Warning System Strategies 

In an examination across all 11 thresholds, two of the 20 coaches met every threshold, 
five of the 20 met at least nine of the thresholds, and 12 of the 20 coaches met at least seven of 
the thresholds. Table 2 reports the means and standard deviations of the 11 thresholds and the 3 
composites. 

The first four thresholds on Table 2 represent activities related to EWI team meetings, 
data use and access, and talks with teachers. A high percent of coaches (75-90%) met the 
thresholds for three of these activities (teacher conversations, early access to data, and examining 
data weekly). Although 18 of the coaches reported accessing data at least weekly, further 
analysis found that 11 coaches stated they looked at data every day they were at the school. The 
only threshold in this group of four activities that a majority of coaches in this group did not 
meet was team meetings. Only 9 of the 20 coaches held meetings every two to three weeks 
focused on students with early warning indicators. Seven coaches reported holding early 
warning team meetings monthly. Only two coaches held meetings less than monthly. 


The next set of seven thresholds on Table 2 represents activities related to class-wide, 
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group, and individual student interventions. Of these interventions, the most widely used were 
classroom talks (70%), attendance incentives (75%), and talking with students (75%). Some 
coaches mentioned meeting with students every time they were in the building. The least-used 
interventions were teacher mentoring (25%), student groups (35%), and more than one report 
card conference (35%). Although only 5 coaches reported consistent use of report card 
conferences over the year, 18 of the 20 coaches reported at least one report card or progress 
report conference. 

The three average scores can be found at the bottom of Table 2. The total Early Warning 
System (TOTAL) score ranged from .09 to 1.00, the Team and Data (TEAM) score ranged from 
0.00 to 1.00, and the Use of Interventions INTERVENTION) composite fidelity score ranged 
from 0.00 to 1.00. Of the three total scores, the team score (average of thresholds 1—4) had the 
highest overall mean (.70) followed by the total score (.577), then the intervention score (.507). 
Correlations among the scores showed significant high correlations between total and team 
scores (.819) and total and intervention scores (.936), which was unsurprising since the team and 
intervention scores were a subset of items that comprised the total score. However, the team and 
intervention scores were also significantly correlated at .565. In examining a scatterplot of the 
relationship between the team and intervention variables, we saw that the six promotion coaches 
with high intervention scores (greater than 0.6) also had high team scores (greater than 0.6). 
However, the twelve coaches with high team scores (greater than 0.6) did not necessarily have 
high intervention scores (from 0.29 to 1.00); six coaches had high team scores but low (less than 
0.6) intervention scores. None of these three scores correlated significantly to school 
characteristics of percent minority students, percent of students receiving free or reduced meals, 


or number of ninth grade students in the school. 
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Finally, we examined the differences among coaches who met, did not meet, or exceeded 
the 20 hour per week standard. Table 3 shows the percentage of coaches meeting each of the 11 
thresholds and the three average scores for each of these three groups. As can be seen, most 
coaches who worked at least 20 hours per week had higher scores than those who worked less 
than 20 hours per week. Those who worked more than 20 hours per week had higher scores on 
most of the thresholds than the other two groups of coaches. One-way analysis of variance 
results show that there were only significant differences among groups on the thresholds relating 
to teacher conversations, examining data weekly, talks with students, and the team score. There 
was a marginally significant finding for total early warning system score. Fisher’s Least 
Significant Difference revealed significant differences between coaches who worked 20 hours a 
week or more than 20 hours per week compared to those who worked less than 20 hours a week. 
There were no significant differences between those who worked 20 hours a week and those who 
worked more than 20 hours a week. 
Impact of Early Warning Systems on Attendance and Course Performance 

Attendance. First time ninth graders had better attendance (16% of students with 
attendance less than 90%) than did all ninth graders including repeaters (20% of students with 
attendance less than 90%), as show in Table 4. Findings from multilevel logistic regressions 
predicting the two attendance scores (above 95% or 90% attendance) for all ninth grade students 
and for just first time ninth grade students, totaling 12 separate regressions, can be found in the 
left columns of Table 5. Results show that the team, intervention, and total scores were not 
significant predictors of attendance of 95% or greater. However, the intervention and total 
scores predicted attendance of 90% or greater for all ninth grade students (p < .05). The 


relationship for first-time ninth graders was significant only at p < .10. 
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More specifically, the odds ratio indicates that the odds of a ninth grade student at a 
school with a total score of 1 (all 11 components in place) having 90% or higher attendance were 
2.6 to 1, compared to a student at a school with a total intervention score of 0 (no components in 
place). Therefore, students at the highest implementation schools are more than 2% times as 
likely to be on track in attendance as students at schools with no implementation at all. The odds 
ratio for the intervention score is similar, but the odds ratio for the team score (just the data and 
EWI meeting components) is not significantly different than 1 (even odds). 

Findings from six separate HLM regression analyses predicting the continuous variable 
percent attendance for all ninth grade students and for first time ninth grade students can also be 
found in Table 5. The intervention score significantly predicted percent ninth grade attendance 
for both first time ninth grade students (p < .05) and all ninth grade students (p < .01). The total 
score significantly predicted percent ninth grade attendance for all ninth grade students (p < .05). 

Course passing. First time ninth graders had better course performance (85% of students 
failing one or fewer courses) than did all ninth graders including repeaters (80% of students 
failing one or fewer courses), as shown in Table 4. Findings from multilevel logistic regressions 
predicting the two course passing scores (passed all courses or failed no more than one course) 
for all ninth grade students, and for just first time ninth grade students, totaling 12 separate 
regressions, can be found in the left columns of Table 6. The intervention and total scores 
significantly predicted both passing all courses and failing no more than one course for all ninth 
grade students (p < .05). Several other relationships were significant only at p < .10, including 
the team score on failing no more than one course for the full group of ninth grade students. 
None of the relationships from the multilevel logistic regression analyses were significant at p < 


.05 for the somewhat smaller group of first-time ninth grade students. 
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Further, the odds ratio indicates that the odds of a ninth grade student at a school with a 
total intervention score of 1 (all 11 components in place) passing all courses were 3.2 to 1, 
compared to a student at a school with a total intervention score of 0 (no components in place). 
Therefore, students at the highest implementation schools were more than 3 times as likely to 
pass all their courses as students at schools with no implementation at all. The odds ratio for the 
intervention score is similar, but the odds ratio for the team score (just the data and EWI meeting 
components) is not significantly different than 1 (even odds). The odds ratios are even higher for 
the less stringent outcome of failing no more than one course in ninth grade. 

In addition, findings from six separate HLM regression analyses predicting the 
continuous variable percent credits earned for all ninth grade students and for first time ninth 
grade students can also be found on Table 6. The intervention and total scores significantly 
predicted percent ninth grade credits earned for both first time ninth grade students (p < .05) and 
all ninth grade students (p < .01). The team score significantly predicted percent ninth grade 
credits earned for all ninth grade students (p < .05). 

Discussion 

Although national findings indicate that over half of public high schools in the U.S. now 
implement some type of early warning systems using early warning indicator data to identify 
students who are at risk of dropping out of school (U.S. Department of Education, 2016), just 
identifying students at risk for dropping out may not be enough. These systems could be 
improved with an additional intervention process, as are used in early warning intervention 
systems such as the Early Warning Intervention and Monitoring System (EWIMS) (Faria et al., 
2017) and the “Early Warning Intervention (EWI) Team” model (Davis et al., 2017). 


The analyses detailed in this paper examined the use of these separate system 
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components within an Early Warning Indicator and Intervention system, including a process to 
monitor data on indicators (particularly attendance, behavior, and course grades) that students 
may be falling off track to graduation, and an emphasis on providing appropriate interventions to 
students who were struggling. As previous research has shown, fidelity to an early warning 
indicator and intervention system may be difficult to maintain (Faria et al., 2017). 

The current study examined the extent to which a part-time promotion coach supporting 
the DN early warning indicator team could help the model be implemented with fidelity and to 
what extent fidelity was associated with student outcomes. Findings indicate variability in the 
frequency and scale of the interventions initiated and monitored by the promotion coach and 
EWI team. However, promotion coaches in the current system were held to a less stringent 
standard than that required by the seven-step process of the EWIMS; instead, they were directed 
only to fulfill basic requirements including 1) holding EWI meetings every two to three weeks, 
2) examining data weekly, and 3) providing grade-wide and student level interventions; 
otherwise, coaches were encouraged to adapt the system to their particular schools and students. 
Although most coaches examined data weekly and led interventions, many did not hold meetings 
every two to three weeks. Many coaches had to cope with the time provided by the school, 
which in many cases meant longer meetings held monthly. Some coaches were also better than 
others at implementing interventions, whether because of their prior experiences or thanks to 
greater freedom provided by the administration. 

Our analyses also examined to what extent separate system components (data 
monitoring/team meetings and the various interventions) related to student outcomes. The 
findings suggest that the implementation of EWI interventions may be more strongly related to 


improved student attendance and course passing than simply examining data and holding EWI 
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meetings. The average intervention score correlated significantly with two measures of 
attendance and all three measures of course passing. Students at schools where interventions 
were implemented at higher levels tended to have significantly lower levels of chronic absence 
and were significantly more likely to stay on track in earning credits by passing their ninth grade 
courses, even when controlling for past attendance. The intervention score was just as useful a 
predictor of student outcomes as was the total implementation score (which combined the data 
monitoring and regular team meetings with the various interventions). 

The team score, focused specifically on the regular examination of student data and 
regular meetings with teachers to discuss struggling students and plan interventions, was not a 
significant predictor of student outcomes in attendance and course performance, except that it 
significantly predicted the percent of credits earned when the entire sample of ninth grade 
students was examined. However, one cannot conclude from these analyses that monitoring data 
and team meetings are not an important part of the initiative, as the pattern of implementation 
among the 20 study schools did not include any schools that scored high on interventions and 
low on the team data monitoring component. Further, the limited sample size precludes 
explicitly testing this. In addition, it should be noted that interventions could only be put in place 
using the information accrued through data monitoring and planned during the team meetings. 
Therefore, without data monitoring and team meetings, the interventions would probably have 
been less likely to occur or less effective. 

Although this implementation study does not provide any causal evidence, it does appear 
that data monitoring and team meetings may not be sufficient, in and of themselves, to have an 
impact on student outcomes. These data suggest that certain activities of a part-time promotion 


coach, particularly directing and monitoring interventions for struggling students, are related to 
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student attendance and course performance. Further, these data suggest that a diverse range of 
interventions, as indicated by the total intervention score averaging use of seven different 
interventions, is related to improved attendance and course performance. These findings suggest 
that those implementing early warning systems should make sure that teachers and counselors be 
provided time and space to plan and implement interventions. 

Study results also indicated that the relationship between the interventions and student 
outcomes was more likely to be significant for the full group of ninth grade students (including 
repeat ninth graders) than for the group of just first-time ninth graders. Data from coaches’ logs 
and interviews suggest that coaches paid particular attention to ninth graders who were 
struggling most and had the greatest number of indicators early in the year; these also tended to 
be repeat ninth graders. One of the challenges in implementation of early warning systems 
appears to be ensuring that interventions occur early enough to ensure that problems in 
attendance and failing grades are prevented for first-time ninth graders, rather than addressed 
after they have occurred. This will require careful attention to attendance in the first couple of 
months in the school year, and to Quarter 1 grades, to give students enough time to recover from 
a bad first quarter and still pass their classes. 

This study also suggests that students with severe problems who have already failed ninth 
grade have the potential to consume much of the energy of the staff available to intervene. It is 
critically important to help schools devote resources and energy to doing a good job of 
preventative intervention, rather than waiting to help students recover from failure after it has 
already had a negative impact on their chances of graduating on time. 

Some may question how many hours a week are necessary for a coach to meet the 


expectations of convening a team bi-weekly, examine data, and support direct interventions. The 
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data from this study indicated that no significant differences between the coaches who worked at 
least 20 hours a week and those who worked more than 20 hours. However, coaches who 
worked less than 20 hours per week were less likely than the other groups to meet the 
expectations. This suggests that at least a half-time (20 hours/week) staff member is needed to 
implement the model with fidelity. Because the study did not include comparisons of full-time 
and part-time coaches, we cannot draw any conclusions about how much more effective coaches 
working a 40-hour week might have been. However, it is likely that differences among the 
coaches’ ability to meet the thresholds could also relate to their prior working experience and the 
working relationship with the school administration and staff that enabled them to schedule team 
meetings and implement interventions. 

Researchers and practitioners have already made great strides in determining the early 
warning indicators and establishing early warning systems to identify students in need of 
intervention. The work ahead lies in ensuring that students actually receive needed interventions 
to help them turn around patterns of poor attendance and poor classroom performance. The 
findings of the current study indicate that an early warning system may not be effective without 
an organized intervention plan. Educators must identify students in need of support and work as 
a team to select and implement interventions to help students back on the path to graduation. 
Table 7 provides a summary of the components of the early warning system discussed above for 
school personnel to consult in planning implementation of such a system in their own context. 
Elsewhere, we provide a how-to guide for schools to use in implementing an early warning 
intervention system (Davis, 2012). Discovering the barriers to implementing effective 
interventions and the necessary steps for addressing those barriers is the next agenda item for 


researchers focused on improving high school outcomes. 
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Table 1 


List of Early Warning System Characteristics and Evaluation Thresholds 


EWS Component 


Source of 
Information 


Threshold 


1. Biweekly EWI Meetings 


2. Teacher Conversations 


3. Early Access to Data 


4, Examine Data Weekly 


5. Mentoring to Teachers 


6. Classroom Talks 


7. Report Card Conference 


Interview and 
weekly coach 
logs 


Weekly coach 
logs 


Interview 


Interview 


Weekly coach 
logs 


Interview and 
weekly coach 
logs 


Interview and 
weekly coach 
logs 


Schools were coded: (1) if the coach held EWI 
team meetings more than monthly (every two to 
three weeks) and those meetings were dedicated 
to matching interventions to a focus group of 
students; (0) if the coach did not. 


Schools were coded: (1) if the coach had two or 
more conversations with teachers noted in their 
weekly logs; (0) if the coach did not. 


Schools were coded: (1) if the coach had 
administrator level access within the first month 
of the school year; (0) if the coach did not..! 


Schools were coded: (1) if the coach looked at 
data at least weekly; (0) if the coach did not. 


Schools were coded: (1) if the coach mentioned 
at least once in the log talking with a teacher or 
a group of teachers on how to improve their 
teaching;(0) if the coach did not. 


Schools were coded: (1) if the coach mentioned 
talking with a class on the importance of 
attendance, behavior, or course performance at 
least once in the weekly logs or in the 
interview; (0) if the coach did not. 


Because nearly every coach completed at least 
one report card conference, schools were coded 
(1) for report card conferences if it could be 
determined that more than one report card or 
progress report conference was held during the 
year; (0) if this did not occur more than once. 


1 Note that coding was conducted on data from the second year of implementation and not on data from the first year 
when implementation began later in the year. 
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8. Attendance Incentive 


9. Student Groups 


10. Talk with Parents 


11. Talk with Students 


Total Early Warning System 
(TOTAL) 


Team and Data (TEAM) 


Use of Interventions 
(INTERVENTION) 


Interview and 
weekly coach 
logs 


Interview and 
weekly coach 
logs 


Interview and 
weekly coach 
logs 


Interview and 
weekly coach 
logs 


Schools were coded: (1) if the coach provided 
some type of attendance incentive; (0) if the 
coach did not. 


Schools were coded: (1) if the coach initiated a 
group of students that met more than once 
during the year; (0) if the coach did not. 


Schools were coded: (1) if the coach mentioned 
calling or meeting with parents more than once 
during the year; (0) if the coach did not. 


Schools were coded: (1) if the coach mentioned 
meeting with students at least weekly; (0) if the 
coach mentioned meeting with students less 
than weekly. 


An average of all threshold scores related to the Use of an Early 
Warning System (Thresholds 1-11) 


An average of all threshold scores related to the Use of an Early 
Warning Team and Data (Thresholds 1-4) 


An average of all threshold scores related to the Use of 
Interventions (Thresholds 5—11) 
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Table 2 


Promotion Coach Use of Early Warning System Actions and Interventions 


Threshold 


9: 


. Biweekly EWI Meetings 
. Teacher Conversations 

. Early Access to Data 

. Examine Data Weekly 

. Teacher Mentoring 

. Classroom Talks 

. Report Card Conference 


. Attendance Incentive 


Student Groups 


10. Talk with Parents 


11. Talk with Students 


Total Early Warning System (TOTAL) 
Early Warning Team and Data (TEAM) 


Use of Interventions INTERVENTION) 


Mean 


0.450 


0.750 


0.750 


0.900 


0.250 


0.700 


0.350 


0.750 


0.350 


0.400 


0.750 


0.577 


0.700 


0.507 


SD 


0.510 


0.444 


0.470 


0.308 


0.444 


0.470 


0.489 


0.444 


0.489 


0.503 


0.444 


0.245 


0.288 


0.268 


Number of 
Coaches 
Implementing 


) 
15 
15 
18 

fs) 
14 

7 


15 
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Table 3 


Promotion Coach Use of Early Warning System Actions and Interventions 


Threshold Less than Twenty More than 

20Hours Hours 20Hours ANOVA 

Sig. 

N=3 N=11 N=6 
1. Biweekly EWI Meetings 0.00 0.45 0.67 ns 
2. Teacher Conversations 0.00 0.91 0.83 001 
3. Early Access to Data 0.67 0.55 1.00 ns 
4, Examine Data Weekly 0.33 1.00 1.00 .000 
5. Teacher Mentoring 0.00 0.27 0.33 ns 
6. Classroom Talks 0.67 0.64 0.83 ns 
7. Report Card Conference 0.33 0.27 0.50 ns 
8. Attendance Incentive 0.67 0.73 0.83 ns 
9. Student Groups 0.00 0.91 0.83 ns 
10. Talk with Parents 0.00 0.45 0.50 ns 
11. Talk with Students 0.00 0.91 0.83 001 
Total Early Warning System (TOTAL) 0.33 0.75 0.75 .096 
Early Warning Team and Data (TEAM) 0.33 0.60 0.62 .006 
Use of Interventions INTERVENTION) 0.33 0.65 0.67 ns 
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Table 4 


Means and Standard Deviation of Dependent Variables for First Time Ninth Graders and All 


Ninth Graders 


First Time All Ninth 
Ninth Graders Grade Students 
(N = 2,815) (N = 3,107) 
Mean SD Mean SD 
Attendance 95% or Higher 0.584 0.493 0.552 0.497 
Attendance 90% or Higher 0.835 0.371 0.797 0.402 
Percent of Days Attended 93.816 6.839 92.832 8.586 
Passed All Courses 0.704 0.456 0.651 0.477 
Failed 1 or 0 Courses 0.848 0.359 0.798 0.402 
Proportion of Ninth Credits 0.924 0.162 0.894 0.204 
Earned 
Previous Year’s Attendance 94.308 5.371 93.365 7.226 
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Table 5 


Hierarchical Modeling Results of Team, Intervention, and Total Scores Predicting More than 95% and 90% Ninth 
Grade Attendance, and Percent of Ninth Grade Attendance, While Controlling for Previous Year’s Attendance. 


95% Attendance or Higher 90% Attendance or Higher % Attendance 
Coefficient (SE) Odds Ratio Coefficient (SE) Odds Ratio Coefficient (SE) 


First Time Ninth Grade 


Team 0.098 (0.355) 1.103 0.345 (0.148) 1.412 0.521 (0.815) 

Intervention 0.211 (0.376) 1.235 0.836T (0.408) 2.306 1.914* (0.761) 

Total 0.209 (0.411) 1.232 0.8087 (0.459) 2.244 1.716f (0.875) 
All Ninth Grade 

Team 0.199 (0.33) 1.221 0.493 (0.382) 1.638 1.210 (0.964) 

Intervention 0.274 (0.350) 1.315 0.927* (0.375) 2.528 2.796** (0.854) 

Total 0.307 (0.382) 1.359 0.949* (0.417) 2.582 2.730* (0.983) 


Note. TSignificant at the .10 level. *Significant at the .05 level. **Significant at the .01 level. 
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Table 6 


Hierarchical Modeling Results of Team, Intervention, and Total Scores Predicting Passing All or Failing Only 
One Course, and Percent of Ninth Grade Credits Earned, While Controlling for Previous Year’s Attendance. 


Passed All Courses Failed 1 or 0 Courses % Credits Earned 
Coefficient (SE) Odds Ratio Coefficient (SE) _Odds Ratio Coefficient (SE) 


First Time Ninth Grade 


Team 0.466 (0.474) 1.594 0.889 (0.614) 2.432 0.053 (0.027) 

Intervention 0.8697 (0.471) 2.383 1.231T (0.623) 3.424 0.070* (0.027) 

Total 0.890 (0.523) 2.434 1.3777 (0.677) 3.962 0.080* (0.029) 
All Ninth Grade 

Team 0.707 (0.492) 2.028 1.0817 (0.614) 2.947 0.089* (0.035) 

Intervention 1.086* (0.490) 2.963 1.430* (0.629) 4.179 0.105** (0.036) 

Total 1.176* (0.536) 3.243 1.627* (0.672) 5.090 0.124** (0.037) 


Note. TSignificant at the .10 level. *Significant at the .05 level. **Significant at the .01 level. 
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Table 7 


Key Components of an Early Warning Indicator and Intervention System 


EWI Team Coach Practices 


Regular access to Early Warning Indicator Data (Attendance, Behavior, Course Grades) 
Examine data weekly 

Hold bi-weekly EWI team meetings with teachers, counselors, student support staff 
Hold informal conversations with teachers regularly 


Important Interventions to Implement to Help Struggling Students 


Student-focused 
School-wide attendance initiatives 
Classroom talks to students about how to stay on-track 
Quarterly report card conferences 
Meetings with small groups of struggling students 
Talks with individual students 
Teacher-focused — Peer mentoring of teachers to encourage restorative intervention practices 


Parent-focused — Contact with parents of struggling students 


